Feat: custom security groups#4027
Open
james-boydell wants to merge 6 commits into
Open
Conversation
… GCP, OCI backends Adds a way to point dstack at a pre-existing, user-managed network security resource instead of the one dstack creates and manages automatically (which always opens SSH to 0.0.0.0/0). Useful for private-network setups (e.g. a VPC only reachable via a Tailscale subnet router). - AWS: security_group_id (backend config) - Azure: network_security_group (backend config) - OCI: network_security_group_id (backend config) - GCP: create_firewall_rules: false (backend config; GCP firewall rules are VPC-wide, not an attachable per-instance resource, so there is no fleet-level override for GCP) - AWS/Azure/OCI also support a fleet/run-level override via the security_group profile property (mirrors the existing reservation field), which takes precedence over the backend config default. When a custom security group is configured, dstack attaches it as-is and never adds, removes, or modifies rules on it. Users are responsible for SSH reachability and, for multi-node clusters, for allowing traffic between instances in the group. Gateway security groups/NSGs (AWS, Azure, OCI) are unaffected by this change and keep dstack's existing auto-managed behavior.
security_group_id / network_security_group / network_security_group_id were flat single-value fields, but security groups and NSGs are scoped per VPC/region, so they only worked for single-region backends. Fixed to mirror the existing vpc_ids/subnet_ids/subnet_ids_per_region conventions: - AWS: security_group_name (a name that must exist in every region's VPC — AWS allows reusing the same group name across regions) and security_group_ids (an explicit region -> ID map for when names differ). - Azure: network_security_group_ids (location -> NSG name map). Azure NSG names are unique per resource group regardless of location, so a single name can never cover more than one region. - OCI: network_security_group_ids (region -> NSG OCID map). Regions/locations not covered by the mapping fall back to dstack's auto-created security group, so partial custom-SG adoption across regions works too. GCP is unaffected (its firewall rules are VPC-wide, not per-region).
An independent review of the custom-security-group feature (previous two commits) found several real gaps. Fixed all of them: - security_group was dropped when a run provisioned a new instance into an existing fleet (only the fleet-apply path honored it). Fixed by threading security_group through the Requirements pipeline exactly like reservation already is (Requirements.security_group, combine_fleet_and_run_profiles/ combine_fleet_and_run_requirements, and sourcing run_job's InstanceConfiguration from job.job_spec.requirements.security_group instead of the run's raw profile). - security_group was silently ignored when an offer resolved to a backend that doesn't support it (e.g. GCP). offers.py now narrows backend_types to BACKENDS_WITH_SECURITY_GROUP_SUPPORT when security_group is set, mirroring the existing reservation filtering. - AWS: the configurator forbade combining security_group_name with security_group_ids, but compute.py implements (and docs/tests described) a fallback from ids to name - the combination is now allowed. Also added validation catching region-key typos in security_group_ids, and a clearer ComputeError instead of a confusing NoCapacityError retry loop when a configured security group doesn't exist in the target VPC. - Azure: renamed network_security_group_ids to network_security_group_names since the values are NSG names (not IDs) scoped to the backend's resource_group, and added region-key typo validation. - GCP: create_firewall_rules no longer disables the gateway firewall rule, matching AWS/Azure/OCI where gateway security resources are always auto-managed regardless of the custom-security-group settings. - OCI: the shared subnet has no security_list_ids, so it inherits the VCN's permissive default security list (SSH open to 0.0.0.0/0, allow-all egress). Since OCI evaluates security lists and NSGs as a union of allows, a custom NSG could not actually restrict anything. Fixed by routing custom-NSG instances into a separate, dedicated VCN/subnet with no security list, so the NSG becomes the sole security boundary. The default VCN/subnet used by auto-managed-NSG instances is completely untouched. Also added region-key typo validation and corrected the docs to accurately attribute default SSH exposure to the security list, not the NSG.
- run_job(): source reservation/security_group from the `requirements` parameter (already fleet+run-combined) instead of job.job_spec.requirements (run-only). Without this, a run provisioning new capacity into a fleet with a fleet-level security_group/reservation would silently ignore it - the same latent bug reservation already had, inherited by security_group. - Azure create_gateway(): use get_gateway_network_security_group_name (the dedicated, always-created gateway NSG) instead of get_default_network_security_group_name. The default/per-location instance NSG can now be skipped when network_security_group_names covers that location, which would have broken gateway provisioning since it was referencing an NSG that might not exist. - OCI: redesign custom-NSG networking. The previous "separate restricted VCN" approach is fundamentally broken - OCI network security groups are VCN-scoped, so a user's NSG can never be attached to an instance in a different VCN than the one the NSG lives in. Fixed by using a single shared subnet for all instances (default-NSG and custom-NSG alike), with no OCI security list attached. dstack's auto-managed NSG now carries explicit SSH ingress and all-egress rules to compensate for the removed security list; custom NSGs remain fully hands-off, per the feature's contract. Existing subnets are migrated in place (security list detached) since the subnet is dstack-owned infrastructure, not a user-supplied resource. Updated docs to clarify a custom NSG must live in dstack's own default VCN. Regression: 388 core backend tests + base/azure/oci targeted suites + server routers/services (requirements, offers, fleets, runs, backends) all passing. Pre-existing unrelated failures (verda/vastai/nebius modules not installed in this environment) left untouched.
r4victor
reviewed
Jul 14, 2026
Comment on lines
+246
to
+247
| By default, `dstack` creates and manages its own security group per project (opening SSH to `0.0.0.0/0` | ||
| and allowing all traffic within the group so multi-node clusters work out of the box). |
Collaborator
There was a problem hiding this comment.
Please try to use the same wording across backends for consistency (with backend-specific terminology) . Something like this:
By default, dstack creates and manages its own security group per project that allows: SSH ingress (TCP port 22) from 0.0.0.0/0, unrestricted egress, and all traffic within the group so that multi-node clusters work out of the box.
Contributor
Author
There was a problem hiding this comment.
Made an attempt to update the wording/phrasing. Feel free to modify as you see fit!
Collaborator
|
@james-boydell, by default I understand that security groups are complimentary to "no public ip" and can be useful to limit the connections from entire VPCs/peered VPCs to specific IPs, control the blast radius, meet the compliance etc. So PR makes sense 100%. Can you expand why custom security groups are needed for your particular use case? |
…s backends
Addresses PR review feedback: use the maintainer's suggested template
consistently across AWS, Azure, and OCI ('creates and manages its own
<security group/NSG> per project that allows: SSH ingress (TCP port 22)
from <source>, unrestricted egress, and all traffic within the <group/
network> so that multi-node clusters work out of the box'), swapping only
backend-specific terminology. GCP keeps a distinct structure since it's a
VPC-wide firewall rule rather than a per-instance-attachable resource.
Also fixes accuracy issues in the 'responsibility for a custom one'
paragraphs found during review:
- GCP: the firewall rule is per VPC network, not per dstack project;
documents that dstack creates neither the instance nor gateway firewall
rule when a shared VPC is configured, and corrects the gateway rule's
ports (22/80/443, not just HTTP/HTTPS).
- OCI: converted the bulleted responsibility list to prose matching the
AWS/Azure phrasing.
- Azure: notes that a custom NSG only needs an explicit SSH-ingress rule,
since Azure's own default NSG rules already permit egress and intra-VNet
traffic on any NSG unless overridden — kept in the 'what you're
responsible for' paragraph, while the opening 'default behavior'
sentence matches the uniform template per the maintainer's request.
- AWS: removed a doubled 'from ... from'.
Contributor
Author
The immediate need is to meet internal compliance. It's understood that using instances without public IPs limits public access which has allowed us to make an internal exception for the time being. However we'd like to clean this up and comply with principle of least privilege and minimize the mute lists for our internal scanners. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Custom security groups / NSGs / firewall rules
Adds support for using a pre-existing security group instead of the one dstack creates automatically, so instances can run in private VPCs without an internet-exposed rule.
New security_group profile property (project-level default via backend config, overridable per fleet/run).
Supported on AWS (security_group_name/security_group_ids), Azure (network_security_group_names), and OCI (network_security_group_ids), each with per-region/location mapping. GCP gets a create_firewall_rules: false opt-out instead, since GCP firewall rules are VPC-wide.
When a custom security group is set, dstack never adds, removes, or modifies its rules — you're fully responsible for ingress/egress.
Docs updated in mkdocs/docs/concepts/backends.md.